A guide to performing a peer review of randomised controlled trials

Peer review of journal articles is an important step in the research process. Editors rely on the expertise of peer reviewers to properly assess submissions. Yet, peer review quality varies widely and few receive training or guidance in how to approach the task. This paper describes some of the main steps that peer reviewers in general and, in particular, those performing reviewes of randomised controlled trials (RCT), can use when carrying out a review. It can be helpful to begin with a brief read to acquaint yourself with the study, followed by a detailed read and a careful check for flaws. These can be divided into ‘major’ (problems that must be resolved before publication can be considered) and ‘minor’ (suggested improvements that are discretionary) flaws. Being aware of the appropriate reporting checklist for the study being reviewed (such as CONSORT and its extensions for RCTs) can also be valuable. Competing interests or prejudices might corrode the review, so ensuring transparency about them is important. Finally, ensuring that the paper’s strengths are acknowledged along with a dissection of the weaknesses provides balance and perspective to both authors and editors. Helpful reviews are constructive and improve the quality of the paper. The proper conduct of a peer review is the responsibility of all who accept the role

Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers

BACKGROUND: Citation screening is time consuming and inefficient. We sought to evaluate the performance of Abstrackr, a semi-automated online tool for predictive title and abstract screening. METHODS: Four systematic reviews (aHUS, dietary fibre, ECHO, rituximab) were used to evaluate Abstrackr. Citations from electronic searches of biomedical databases were imported into Abstrackr, and titles and abstracts were screened and included or excluded according to the entry criteria. This process was continued until Abstrackr predicted and classified the remaining unscreened citations as relevant or irrelevant. These classification predictions were checked for accuracy against the original review decisions. Sensitivity analyses were performed to assess the effects of including case reports in the aHUS dataset whilst screening and the effects of using larger imbalanced datasets with the ECHO dataset. The performance of Abstrackr was calculated according to the number of relevant studies missed, the workload saving, the false negative rate, and the precision of the algorithm to correctly predict relevant studies for inclusion, i.e. further full text inspection. RESULTS: Of the unscreened citations, Abstrackr’s prediction algorithm correctly identified all relevant citations for the rituximab and dietary fibre reviews. However, one relevant citation in both the aHUS and ECHO reviews was incorrectly predicted as not relevant. The workload saving achieved with Abstrackr varied depending on the complexity and size of the reviews (9 % rituximab, 40 % dietary fibre, 67 % aHUS, and 57 % ECHO). The proportion of citations predicted as relevant, and therefore, warranting further full text inspection (i.e. the precision of the prediction) ranged from 16 % (aHUS) to 45 % (rituximab) and was affected by the complexity of the reviews. The false negative rate ranged from 2.4 to 21.7 %. Sensitivity analysis performed on the aHUS dataset increased the precision from 16 to 25 % and increased the workload saving by 10 % but increased the number of relevant studies missed. Sensitivity analysis performed with the larger ECHO dataset increased the workload saving (80 %) but reduced the precision (6.8 %) and increased the number of missed citations. CONCLUSIONS: Semi-automated title and abstract screening with Abstrackr has the potential to save time and reduce research waste

Exploration of the methodological quality and clinical usefulness of a cross-sectional sample of published guidance about exercise training and physical activity for the secondary prevention of coronary heart disease

Background Clinicians are encouraged to use guidelines to assist in providing evidence-based secondary prevention to patients with coronary heart disease. However, the expanding number of publications providing guidance about exercise training may confuse cardiac rehabilitation clinicians. We therefore sought to explore the number, scope, publication characteristics, methodological quality, and clinical usefulness of published exercise-based cardiac rehabilitation guidance. Methods We included publications recommending physical activity, exercise or cardiac rehabilitation for patients with coronary heart disease. These included systematically developed clinical practice guidelines, as well as other publications intended to support clinician decision making, such as position papers or consensus statements. Publications were obtained via electronic searches of preventive cardiology societies, guideline databases and PubMed, to November 2016. Publication characteristics were extracted, and two independent assessors evaluated quality using the 23-item Appraisal of Guidelines Research and Evaluation II (AGREE) tool. Results Fifty-four international publications from 1994 to 2016 were identified. Most were found on preventive cardiology association websites (n = 35; 65%) and were freely accessible (n = 50; 93%). Thirty (56%) publications contained only broad recommendations for physical activity and cardiac rehabilitation referral, while 24 (44%) contained the necessary detailed exercise training recommendations. Many were labelled as “guidelines”, however publications with other titles (e.g. scientific statements) were common (n = 24; 44%). This latter group of publications contained a significantly greater proportion of detailed exercise training recommendations than clinical guidelines (p = 0.017). Wide variation in quality also existed, with ‘applicability’ the worst scoring AGREE II domain for clinical guidelines (mean score 53%) and ‘rigour of development’ rated lowest for other guidance types (mean score 33%). Conclusions While a large number of guidance documents provide recommendations for exercise-based cardiac rehabilitation, most have limitations in either methodological quality or clinical usefulness. The lack of rigorously developed guidelines which also contain necessary detail about exercise training remains a substantial problem for clinicians

Patients' expectations of the benefits and harms of treatments, screening, and tests a systematic review

IMPORTANCE Unrealistic patient expectations of the benefits and harms of interventions can influence decision making and may be contributing to increasing intervention uptake and health care costs. OBJECTIVE To systematically review all studies that have quantitatively assessed patients' expectations of the benefits and/or harms of any treatment, test, or screening test. EVIDENCE REVIEW A comprehensive search strategy was used in 4 databases (MEDLINE, Embase, Cumulative Index to Nursing and Allied Health Literature, PsycINFO) up to June 2013, with no language or study type restriction.We also ran cited reference searches of included studies and contacted experts and study authors. Two researchers independently evaluated methodological quality and extracted participants' estimates of benefit and harms and authors' contemporaneous estimates. FINDINGS Of the 15 343 records screened, 36 articles (from 35 studies) involving a total of 27 323 patients were eligible. Fourteen studies focused on a screen, 15 on treatment, 3 a test, and 3 on treatment and screening. More studies assessed only benefit expectations (22 [63%]) than benefit and harm expectations (10 [29%]) or only harm (3 [8%]). Fifty-four outcomes (across 32 studies) assessed benefit expectations: of the 34 outcomes with overestimation data available, the majority of participants overestimated benefit for 22 (65%) of them. For 17 benefit expectation outcomes, we could not calculate the proportion of participants who overestimated or underestimated, although for 15 (88%) of these, study authors concluded that participants overestimated benefits. Expectations of harm were assessed by 27 outcomes (across 13 studies): underestimation data were available for 15 outcomes and the majority of participants underestimated harm for 10 (67%) of these. A correct estimation by at least 50% of participants only occurred for 2 outcomes about benefit expectations and 2 outcomes about harm expectations. CONCLUSIONS AND RELEVANCE The majority of participants overestimated intervention benefit and underestimated harm. Clinicians should discuss accurate and balanced information about intervention benefits and harms with patients, providing the opportunity to develop realistic expectations and make informed decisions

Exercise training characteristics in cardiac rehabilitation programmes: A cross-sectional survey of Australian practice

Introduction Exercise training is a core component of cardiac rehabilitation (CR), however, little information exists regarding the specific exercise interventions currently provided for coronary heart disease in Australian practice. We aimed to analyse the current status of exercise-based CR services across Australia. Design Cross-sectional survey. Methods Australian sites offering exercise-based CR were identified from publically available directories. All sites were invited by email to participate in an online Survey Monkey questionnaire between October 2014 and March 2015, with reminders via email and phone follow-up. Questions investigated the demographics and format of individual programmes, as well as specific exercise training characteristics. Results 297 eligible programmes were identified, with an 82% response rate. Most sites (82%) were based at hospital or outpatient centres, with home (15%), community (18%) or gym-based options (5%) less common. While CR was most often offered in a comprehensive format (72% of sites), the level of exercise intervention varied greatly among programmes. Most frequently, exercise was prescribed 1–2 times per week for 60 min over 7 weeks. Almost one-quarter (24%) had a sole practitioner supervising exercise, although the majority used a nurse/physiotherapist combination. Low to moderate exercise intensities were used in 60% of programmes, however, higher intensity prescriptions were not uncommon. Few sites (Conclusions While advances have been made towards providing flexible and accessible exercise-based CR, much of Australia's service remains within traditional models of care. A continuing focus on service improvement and evidence-based care should, therefore, be considered a core aim of those providing exercise for CR in order to improve health service delivery and optimise outcomes for patients

Better duplicate detection for systematic reviewers: Evaluation of Systematic Review Assistant-Deduplication Module

BACKGROUND: A major problem arising from searching across bibliographic databases is the retrieval of duplicate citations. Removing such duplicates is an essential task to ensure systematic reviewers do not waste time screening the same citation multiple times. Although reference management software use algorithms to remove duplicate records, this is only partially successful and necessitates removing the remaining duplicates manually. This time-consuming task leads to wasted resources. We sought to evaluate the effectiveness of a newly developed deduplication program against EndNote. METHODS: A literature search of 1,988 citations was manually inspected and duplicate citations identified and coded to create a benchmark dataset. The Systematic Review Assistant-Deduplication Module (SRA-DM) was iteratively developed and tested using the benchmark dataset and compared with EndNote’s default one step auto-deduplication process matching on (‘author’, ‘year’, ‘title’). The accuracy of deduplication was reported by calculating the sensitivity and specificity. Further validation tests, with three additional benchmarked literature searches comprising a total of 4,563 citations were performed to determine the reliability of the SRA-DM algorithm. RESULTS: The sensitivity (84%) and specificity (100%) of the SRA-DM was superior to EndNote (sensitivity 51%, specificity 99.83%). Validation testing on three additional biomedical literature searches demonstrated that SRA-DM consistently achieved higher sensitivity than EndNote (90% vs 63%), (84% vs 73%) and (84% vs 64%). Furthermore, the specificity of SRA-DM was 100%, whereas the specificity of EndNote was imperfect (average 99.75%) with some unique records wrongly assigned as duplicates. Overall, there was a 42.86% increase in the number of duplicates records detected with SRA-DM compared with EndNote auto-deduplication. CONCLUSIONS: The Systematic Review Assistant-Deduplication Module offers users a reliable program to remove duplicate records with greater sensitivity and specificity than EndNote. This application will save researchers and information specialists time and avoid research waste. The deduplication program is freely available online

Poor description of non-pharmacological interventions:Analysis of consecutive sample of randomised trials

Objectives To evaluate the completeness of descriptions of non-pharmacological interventions in randomised trials, identify which elements are most frequently missing, and assess whether authors can provide missing details. Design Analysis of consecutive sample of randomised trials of non-pharmacological interventions. Data sources and study selection All reports of randomised trials of non-pharmacological interventions published in 2009 in six leading general medical journals; 133 trial reports, with 137 interventions, met the inclusion criteria. Data collection Using an eight item checklist, two raters assessed the primary full trial report, plus any reference materials, appendices, or websites. Questions about missing details were emailed to corresponding authors, and relevant items were then reassessed. Results Of 137 interventions, only 53 (39%) were adequately described; this was increased to 81 (59%) by using 63 responses from 88 contacted authors. The most frequently missing item was the “intervention materials” (47% complete), but it also improved the most after author response (92% complete). Whereas some authors (27/70) provided materials or further information, other authors (21/70) could not; their reasons included copyright or intellectual property concerns, not having the materials or intervention details, or being unaware of their importance. Although 46 (34%) trial interventions had further information or materials readily available on a website, many were not mentioned in the report, were not freely accessible, or the URL was no longer functioning. Conclusions Missing essential information about interventions is a frequent, yet remediable, contributor to the worldwide waste in research funding. If trial reports do not have a sufficient description of interventions, other researchers cannot build on the findings, and clinicians and patients cannot reliably implement useful interventions. Improvement will require action by funders, researchers, and publishers, aided by long term repositories of materials linked to publications

A comparison of the performance of seven key bibliographic databases in identifying all relevant systematic reviews of interventions for hypertension

BACKGROUND: Bibliographic databases are the primary resource for identifying systematic reviews of health care interventions. Reliable retrieval of systematic reviews depends on the scope of indexing used by database providers. Therefore, searching one database may be insufficient, but it is unclear how many need to be searched. We sought to evaluate the performance of seven major bibliographic databases for the identification of systematic reviews for hypertension. METHODS: We searched seven databases (Cochrane library, Database of Abstracts of Reviews of Effects (DARE), Excerpta Medica Database (EMBASE), Epistemonikos, Medical Literature Analysis and Retrieval System Online (MEDLINE), PubMed Health and Turning Research Into Practice (TRIP)) from 2003 to 2015 for systematic reviews of any intervention for hypertension. Citations retrieved were screened for relevance, coded and checked for screening consistency using a fuzzy text matching query. The performance of each database was assessed by calculating its sensitivity, precision, the number of missed reviews and the number of unique records retrieved. RESULTS: Four hundred systematic reviews were identified for inclusion from 11,381 citations retrieved from seven databases. No single database identified all the retrieved systematic reviews for hypertension. EMBASE identified the most reviews (sensitivity 69 %) but also retrieved the most irrelevant citations with 7.2 % precision (Pr). The sensitivity of the Cochrane library was 60 %, DARE 57 %, MEDLINE 57 %, PubMed Health 53 %, Epistemonikos 49 % and TRIP 33 %. EMBASE contained the highest number of unique records (n = 43). The Cochrane library identified seven unique records and had the highest precision (Pr = 30 %), followed by Epistemonikos (n = 2, Pr = 19 %). No unique records were found in PubMed Health (Pr = 24 %) DARE (Pr = 21 %), TRIP (Pr = 10 %) or MEDLINE (Pr = 10 %). Searching EMBASE and the Cochrane library identified 88 % of all systematic reviews in the reference set, and searching the freely available databases (Cochrane, Epistemonikos, MEDLINE) identified 83 % of all the reviews. The databases were re-analysed after systematic reviews of non-conventional interventions (e.g. yoga, acupuncture) were removed. Similarly, no database identified all the retrieved systematic reviews. EMBASE identified the most relevant systematic reviews (sensitivity 73 %) but also retrieved the most irrelevant citations with Pr = 5 %. The sensitivity of the Cochrane database was 62 %, followed by MEDLINE (60 %), DARE (55 %), PubMed Health (54 %), Epistemonikos (50 %) and TRIP (31 %). The precision of the Cochrane library was the highest (20 %), followed by PubMed Health (Pr = 16 %), DARE (Pr = 13 %), Epistemonikos (Pr = 12 %), MEDLINE (Pr = 6 %), TRIP (Pr = 6 %) and EMBASE (Pr = 5 %). EMBASE contained the most unique records (n = 34). The Cochrane library identified seven unique records. The other databases held no unique records. CONCLUSIONS: The coverage of bibliographic databases varies considerably due to differences in their scope and content. Researchers wishing to identify systematic reviews should not rely on one database but search multiple databases

Shared decision making and antibiotic benefit-harm conversations: an observational study of consultations between general practitioners and patients with acute respiratory infections

Abstract Background Little research has examined whether shared decision making (SDM) occurs in consultations for acute respiratory infections (ARIs), including what, and how, antibiotic benefits and harms are discussed. We aimed to analyse the extent and nature of SDM in consultations between GPs and patients with ARIs, and explore communication with and without the use of patient decision aids. Methods This was an observational study in Australian general practices, nested within a cluster randomised trial of decision aids (for acute otitis media [AOM], sore throat, acute bronchitis) designed for general practitioners (GPs) to use with patients, compared with usual care (no decision aids). Audio-recordings of consultations of a convenience sample of consenting patients seeing a GP for an ARI were independently analysed by two raters using the OPTION-12 (observing patient involvement in decision making) scale (maximum score of 100) and 5 items (about communicating evidence) from the Assessing Communication about Evidence and Patient Preferences (ACEPP) tool (maximum score of 5). Patients also self-completed a questionnaire post-consultation that contained items from CollaboRATE-5 (perceptions of involvement in the decision-making process), a decisional conflict scale, and a decision self-efficacy scale. Descriptive statistics were calculated for each measure. Results Thirty-six consultations, involving 13 GPs, were recorded (20 for bronchitis, 10 sore throat, 6 AOM). The mean (SD) total OPTION-12 score was 29.4 (12.5; range 4–54), with item 12 (need to review decision) the highest (mean = 3) and item 10 (eliciting patients’ preferred level of decision-making involvement) the lowest (mean = 0.1). The mean (SD) total ACEPP score was 2 (1.6), with the item about discussing benefits scoring highest. In consultations where a decision aid was used (15, 42%), compared to the 21 usual care consultations, mean observer-assessed SDM scores (OPTION-12, ACEPP scores) were higher and antibiotic harms mentioned in all (compared to only 1) consultations. Patients generally reported high decision involvement and self-efficacy, and low decisional conflict. Conclusions The extent of observer-assessed SDM between GPs and patients with ARIs was generally low. Balanced discussion of antibiotic benefits and harms occurred more often when decision aids were used